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Preface 


The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based 
Practice Centers (EPCs), sponsors the development of evidence reports and technology 
assessments to assist public- and private-sector organizations in their efforts to improve the 
quality of health care in the United States. The reports and assessments provide organizations 
with comprehensive, science-based information on common, costly medical conditions and new 
health care technologies and strategies. The EPCs systematically review the relevant scientific 
literature on topics assigned to them by AHRQ and conduct additional analyses when 
appropriate prior to developing their reports and assessments. 

To improve the scientific rigor of these evidence reports, AHRQ supports empiric research 
by the EPCs to help understand or improve complex methodologic issues in systematic reviews. 
These methods research projects are intended to contribute to the research base in and be used to 
improve the science of systematic reviews. They are not intended to be guidance to the EPC 
program, although they may be considered by EPCs along with other scientific research when 
determining EPC program methods guidance. 

AHRQ expects that the EPC evidence reports and technology assessments will inform 
individual health plans, providers, and purchasers and the health care system as a whole by 
providing important information to help improve health care quality. The reports undergo peer 
review prior to their release as a final report. 

We welcome comments on this Methods Research Project. They may be sent by mail to the 
Task Order Officer named below at: Agency for Healthcare Research and Quality, 540 Gaither 
Road, Rockville, MD 20850, or by e-mail to epc @ahrq.hhs.gov. 


Richard Kronick, Ph.D. David Meyers, M.D. 

Director Acting Director 

Agency for Healthcare Research and Quality | Center for Evidence and Practice Improvement 
Agency for Healthcare Research and Quality 


Stephanie Chang, M.D., M.P.H. Elisabeth Kato, M.D., M.R.P. 

Director, Evidence-based Practice Program Task Order Officer 

Center for Evidence and Practice Improvement Center for Evidence and Practice Improvement 
Agency for Healthcare Research and Quality Agency for Healthcare Research and Quality 
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and/or conclusions do not necessarily represent the views of individual Key Informants. 

Key Informants must disclose any financial conflicts of interest greater than $10,000 and any 
other relevant business or professional conflicts of interest. Because of their role as end-users, 
individuals with potential conflicts may be retained. The TOO and the EPC work to balance, 


manage, or mitigate any conflicts of interest. 
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Structured Abstract 


Objectives. To characterize rapid reviews and similar products, to understand the context in 
which rapid products are produced (e.g., end-users and purposes for rapid products), to 
understand methodological guidance and strategies used to make products rapid and describe 
how these differ from systematic review (SR) procedures, and to identify empiric evidence on 
the impact of methodological approaches on their reliability and validity. 


Methods. We searched the literature to identify rapid review methods, empiric evidence on rapid 
review methodology, and methodological guidance. We conducted interviews with members of 
organizations known to produce rapid reviews to characterize the types of rapid products 
produced and to understand the context and uses for rapid products, identify current practices, 
and understand the evolution of their programs and products. 


Results. We identified 36 examples of rapid products produced by 20 organizations with 
production time ranging from 5 minutes to 8 months. 

We categorized rapid products into four groups based on the extent of synthesis: 

(1) *inventories” list what evidence is available, and other contextual information needed to 
make decisions, but do not synthesize the evidence or present summaries or conclusions; 

(2) ’rapid responses” present the end-user with an answer based on the best available evidence 
(usually guidelines or SRs), but do not attempt to formally synthesize the evidence into 
conclusions; (3) ”rapid reviews” perform a synthesis (qualitative and/or quantitative) to provide 
an answer about the direction of evidence and possibly the strength of evidence; (4) “automated 
approaches” use databases of extracted study elements and programming to generate meta- 
analyses in response to user-defined queries. 

Methodological approaches identified for rapid products include: searching fewer databases; 
limited use of grey literature; restricting the types of studies included (e.g., English only, most 
recent 5 years); relying on existing SRs; limiting full-text review; limiting dual review for study 
selection and/or data extraction; limiting data extraction; limiting risk of bias assessment or 
grading; minimal evidence synthesis; providing nominal conclusions or recommendations; and 
limiting external peer review. As the timeframes for products lengthened many limitations were 
lifted; however, there were still restrictions on database searching, inclusion, extent of data 
extraction, and dual review. With lengthened production time, there was more often risk of bias 
assessment, evidence grading, and external peer review. 

Key informant interviews demonstrated that the essence of rapid products differs from that of 
SRs: key differences include the close relationship with the end-user and focus on helping a 
specific end-user make a specific decision in an identified timeframe. Because there may not be 
lead time before the review is needed and the end-user may need the review urgently, 
maintaining a highly skilled staff is critical to organizational readiness to produce rapid reviews. 
Having few and/or narrow questions (e.g., emerging technologies, single interventions, specific 
populations) was also necessary. 

There is almost no empiric evidence directly comparing results of rapid products with SRs. 
One report suggested there may not be any impact; however, it focused on surgical interventions 
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and may not be generalizable to other clinical specialties or health care fields in which rapid 
products or SRs are conducted. 


Conclusions. Rapid products have tremendous methodological variation. Overall, they vary on 
two important dimensions that are captured by the term “rapid review”: the timeframe for 
completion and extent of synthesis. The similarity of rapid products lies in their close 
relationship with the end-user to meet decisionmaking needs in a limited timeframe. The 
following are considerations for creating rapid products: 


products should be developed in the context of identified end-users and their specific 
decisionmaking needs and circumstances; 

a close relationship with the end-user and iterative feedback is essential; 

reliance on existing SRs require methods to summarize and interpret evidence; 

a highly skilled and experienced staff and the capacity to mobilize skilled staff quickly 
are critical; 

restricting scope may be necessary; 

producers and users need to accept modifications to standard SR methods; and 
limitations need to be clearly reported, particularly in terms of potential bias and 
shortcomings of the conclusions. 


Future research evaluating end-user perspectives will complement these findings and provide 
additional considerations for those interested in establishing a rapid response program or 
producing rapid products. 
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Introduction 


Background 


Systematic reviews provide comprehensive and rigorous syntheses of evidence to answer 
clinical and policy questions; however, they are complex undertakings requiring substantial time 
and resources to meet currently accepted standards.’ It is not uncommon for standard systematic 
reviews to take one or even two years to complete. Increasingly, clinicians, policy makers, and 
other decision makers are requesting evidence in shorter timeframes to support their needs;° 
however they still expect transparent, unbiased, and reliable evidence products.” In response, the 
concept of “rapid reviews” has emerged within the arena of evidence synthesis to describe a 
review type that can be completed more quickly than a standard systematic review. 

Given the considerable time and resource requirements to produce systematic reviews that 
comply with endorsed standards,” it is not surprising that the intuitively appealing concept of 
rapid reviews has gained considerable attention recently within communities that produce or rely 
on systematic reviews. For example, sessions on rapid reviews have been offered at the most 
recent annual symposia of The Cochrane Collaboration and Guidelines International Network 
(G-I-N). Further, Cochrane Innovations, a trading company owned by The Cochrane 
Collaboration, has initiated a rapid response program, and a formal methods group is being 
proposed within the Collaboration with a focus on rapid reviews. A 2012 survey of European 
Health Technology Assessment (HTA) and public health organizations suggested that rapid 
reviews are offered by 70 percent of these agencies.” 

This surge in interest poses two challenges to existing systematic review programs. First, 
should rapid reviews be offered, and what components are needed to build a successful program? 
Second, can some rapid review methods be adapted to make standard systematic reviews more 
timely? Answering these questions depends on understanding what methods rapid reviews use to 
achieve their more timely results, and to what extent those methods increase the risk of bias and 
error. Systematic reviews take time because they employ multiple methodologies to ensure that 
all relevant data have been identified and accurately analyzed. These time-consuming 
methodologies have been developed to minimize bias in the final conclusions, and therefore 
eliminating some or all of them may result in biased conclusions. 

Answering these questions is more complicated than it appears. In order to consider offering 
rapid reviews, one first has to be able to define what they are. Previous reviews on the topic of 
rapid reviews describe them as “ill-defined,”® “not well-defined,” lacking a single definition,” 
or “varying widely in terms of the language used to describe them.”'' HTAs may also be rapid, 
and so HTA may be part of the title of such reviews. Given the wide range of terms used to 
capture these short, focused reviews (rapid review, evidence advisory, evidence inventory, 
hotline response, etc.), we use the term “rapid products” in this report to capture the range of 
types of rapid reports. 

Not only the names, but also the timelines vary considerably. Andradas et al. described rapid 
reviews as brief, readable, and usable responses to guide decision making that are typically 
completed within 6 months, which we have also chosen to use as part of our working definition 
of a rapid product.'”!? Previous reviews have focused specifically on rapid HTAs and while 
multiple reviews describe these as taking between 1 and 6 months,”'”’* others described 
products taking as little as 1 week.'> Automated approaches that generate meta-analyses in 
response to user-defined queries, sometimes called an “ultra-rapid review,” are an outlier product 
that takes mere minutes to complete. 


Finally, rapid products employ a wide range of strategies to reduce the time required to 
produce the final report, and an even wider range of combinations of strategies. This includes 
different methods used to identify relevant literature,'”'*'> different approaches to assessing 
quality,'°* and variation in the types of syntheses conducted.*'*!°'® Authors of one review even 
argued that rapid products “defy definitive categorization because of their heterogeneous 
timelines, components, search strategies, and methodologies.”'’ However, given the growing 
interest in these rapid products and the ever-present pressure to produce reliable evidence on 
more topics more quickly, it is imperative that a better understanding be developed. 


Purpose of This Report 


Since 1997, the Agency for Healthcare Research and Quality’s (AHRQ) Evidence-based 
Practice Centers (EPCs) have been recognized as leaders in conducting comprehensive 
systematic reviews that shape health policy, inform national scientific conferences, and assist 
stakeholders in health care decision making. Topics for EPC reviews may be nominated by 
individuals or specific organizations; however, the reports are prepared with the intent of being 
applicable to and usable by a broad audience. Currently, the typical process for the production of 
a systematic review within the EPC Program follows a number of steps including topic 
nomination, topic triage, topic refinement, and completion of the full review, and can take as 
long as 2 years. While these reviews are recognized for their breadth and methodological rigor, 
they cannot be commissioned by stakeholders to provide evidence to support decisions that need 
to be made urgently. 

Given the critical importance of having valid evidence syntheses available in a timely 
manner for use by a myriad of health and health care decision makers, the AHRQ EPC Program 
commissioned a white paper on rapid reviews. The goal of this white paper is to understand the 
range of products that are considered rapid, their intended purposes and the contexts in which 
they are commissioned, and how methods and required resources differ (e.g., how they gain 
efficiency) between products and from standard systematic reviews. Ideally, this white paper will 
also identify available empiric data to judge the impact of specific methodological shortcuts on 
the reliability, validity, and usability of these rapid products. Further, the perceived value of 
these rapid products by end-users, particularly in light of different methodological trade-offs, 
needs to be understood. 

We sought to build on previous work by not simply describing how methods vary between 
rapid products and standard systematic reviews, but by identifying which variations are 
important and how the context in which they are produced influences variations in methods used. 
We also sought to understand the approach to rapid products that different organizations have 
taken particularly in the context of the purpose of the reports, those requesting the reports, the 
types of decisions being informed, the setting and structure of the rapid product developer, and 
the relationship between the developers and users of the reports. The following probing questions 
guided this work: What are the characteristics of rapid products produced by key organizations 
and described in the literature (e.g., purpose, audience, timelines, and personnel)? What 
methodological guidance exists for the conduct of rapid products? What empiric evidence exists 
comparing the results of rapid products with systematic reviews? 


Methods 


General Approach 


We took a two-pronged approach to meet our objectives, combining a review of the literature 
with Key Informant (KI) interviews of key producers of rapid products to develop a broad 
picture of what is considered a rapid review/product, the strategies used to make the products 
rapid, and the context in which such products are produced. A workgroup of members from 
EPCs, the Scientific Resource Center (SRC), and AHRQ participated in twice monthly 
workgroup teleconference calls over the course of 10 months to discuss project direction and 
scope, assign and coordinate tasks, collect and analyze data, and discuss and edit draft 
documents. 


Literature Search 


In October and November 2013, we searched Ovid Medline, Ovid EBM Reviews, Cochrane 
Methodology Register, and SRC Methods Library, and performed a Scopus citation reference 
search and a grey literature search to identify papers about rapid reviews or similar products. 
High value keywords were identified by workgroup members. The final search strategy appears 
in Appendix A. 

We sought to identify literature that: (1) discussed rapid review methods; (2) discussed 
initiatives or programs producing rapid reviews; or (3) provided empiric evidence on rapid 
review methodology. We also identified examples of published rapid reviews. We limited 
inclusion to articles describing rapid products within the health care field. 

Early on we realized that the terminology used to describe rapid products is extremely varied. 
AS we wete interested in understanding broadly how these products are defined and conducted, 
we sought to be inclusive. During our screening we encountered the concept of mini-HTAs. 
Mini-HTAs are described as a “management and decision support tool’”'® and typically consist of 
a form or checklist containing several questions corresponding to the domains of a standard HTA 
report which may be answered based on the literature, a survey, and expert opinion. We excluded 
mini-HTAs from this report because they are a checklist that serves to guide decision making 
and not a method to evaluate evidence. 

Abstracts were dual reviewed by investigators using ABSTRACKR software (available at 
http://abstrackr.cebm.brown.edu). Discrepancies were resolved by a third investigator. Full-text 
articles were independently reviewed by two investigators for inclusion. Articles suggested by 
workgroup members or Key Informants were also reviewed for relevance using the same 
methods. Different groupings of articles emerged during full-text review including: (1) 
background papers on rapid product methods, programs, or contextual factors; (2) previous 
articles of rapid product types; (3) descriptions of rapid products, including the methods used to 
produce them; and (4) empiric data exploring differences resulting from rapid products and 
standard systematic review methods. Groupings are not mutually exclusive. Articles in groups 1 
and 2 provided context and contributed to the introduction and discussion sections of this report. 
Data from articles in groups 3 and 4 were extracted, synthesized and informed the results. 

We reviewed papers that gave an overview or theoretical description of rapid products and 
used these to develop items for a table of products. First, we developed a set of questions to 
describe the products and a framework of dimensions of the process that can be altered to make a 
review more rapid (Table 1). The purpose of this table was to identify which aspects of the 


standard systematic review process are most and least often altered to accelerate the review 
process, whether specific shortcuts are associated with specific features of a review, and what the 
implications are for the utility of the final product. We also identified a set of questions to allow 
us to describe the context in which the reviews are created, such as how long the producer has 
been preparing rapid products, the relationship with the nominator, and the number and 
knowledge-level of the staff. 

Details about reports and programs from articles that described actual rapid reports or 
programs were extracted into the table of examples, and samples of those rapid reports were used 
to crosscheck the information from the articles whenever possible. Studies that empirically 
compared the results of rapid products with standard systematic reviews were extracted and 
analyzed separately. 


Table 1. Dimensions of standard systematic reviews that may be altered in rapid products 





Dimension 





Scope Limit the type of questions (e.g., efficacy only, new technology only, single 
technology only) 





Limit number of questions 





Limit number of studies that can be included 
Comprehensiveness Limit search strategy (e.g., number of databases, grey literature, date, setting, 
language) 





Limit study types included (e.g., existing systematic reviews only, RCTs only) 





Limit textual analysis (e.g., no full-text review, limit number of extraction items) 





Rigor/Quality control Eliminate dual study selection 





Eliminate dual data extraction 





Limit or eliminate internal or external review of final product (e.g., peer review) 





Synthesis Limit or eliminate risk of bias/ quality assessment of individual studies 





Limit or eliminate either quantitative or qualitative analysis 








Limit or eliminate strength/quality of evidence assessments (e.g., using GRADE) 
Conclusions Simplify or eliminate any conclusive statements about the direction of the 
evidence 








Key Informant Interviews 


In parallel with the literature search, we conducted interviews with producers of rapid 
products, in both the public and private sector. We invited 19 Key Informants (KIs) to participate 
in a 60-minute individual telephone interview between January and March of 2014. Seventeen 
interviews were conducted with 18 KIs. The KIs were selected from organizations known by 
members of the workgroup to produce rapid products or offer rapid response services. No 
restrictions were placed with respect to geographic location; however, we were only able to 
conduct interviews with KIs who could communicate in English. We sought KIs from 
organizations around the world that are recognized for work in this area. Based on a comparison 
of KI interviews with the literature review, we are confident that we spoke with most of the 
prominent producers of rapid products in the English language. 

Each KI completed an “EPC Conflict of Interest Disclosure Form” prior to being interviewed 
and no disclosed conflicts precluded participation of any of the invited organizations. KIs were 
interviewed about their methods for and experiences in developing rapid products following an 
interview guide developed by the workgroup (Appendix B). All interviews were digitally 
recorded, transcribed, and reviewed to identify and code themes using NVivo software (QSR 
International). Relevant information on rapid products was compiled in an extraction table. 

The details of specific rapid products described in the interviews were extracted by one 
researcher and reviewed by at least one other and combined with data extracted from the 





literature examples to create a combined table of examples of rapid products. Some of the 
examples are informed only from the literature, some only by the interview, and others by both. 
While this approach has some drawbacks—in general we have less detail about rapid products 
available only through the literature—it allowed us to create a larger and richer set of examples 
than would have been available though one approach alone. For the examples derived from the 
interviews, we sent the tabulated information back to the interviewees for review; we received 
responses from 10 of the organizations. 


Results 


A total of 468 articles were identified through the literature search. After dual review of 
abstracts, 116 articles were pulled for full-text review. Additional articles from workgroup 
members and Key Informants (KIs) were also pulled for full-text review. After full-text review, a 
total of 53 articles were included and were categorized as: background (n=8), review of rapid 
review types (n=12), rapid review/product methods (n=30), and empiric studies evaluating rapid 
review/product methods (n=2). Of the 30 articles initially classified as rapid review/product 
methods, only those that described methods for a specific program or product (n=10) were 
included in the final analysis. A flow diagram of the search process is shown in Appendix C. The 
background articles and reviews of rapid review types were used to provide context for the 
introduction and background sections of this white paper. The other sets of papers are described 
in the sections below, including: 

1. Characterization of Rapid Products and Their Methods (based on the literature review 

and KI interviews) 

2. Comparing Systematic Review and Rapid Review Approaches (based on KI interviews 

and the literature review) 

3. Empiric Evidence Comparing Rapid Products With Systematic Reviews (based on the 

literature review) 


Characterization of Rapid Products and Their Methods 


In this section we integrate information extracted from studies that describe rapid response 
programs or products with information gathered from the KI interviews and, where possible, by 
examining actual examples of the products in question. Based on these two methods of data 
gathering, we identified 36 examples of rapid products produced by 20 organizations. The 
organizations represented were both public and for-profit and had a range of experience with 
rapid products, from several months to 25 years. Most organizations also had well-established 
programs and extensive experience producing standard systematic reviews or HTAs (e.g., 25 
years). Only one organization did not conduct standard systematic reviews or HTAs. The volume 
of reports produced varied by organization and type of product, ranging from 3 to 5 per year to 
300 to 400 per year. 

Most organizations had a multidisciplinary staff that was involved in producing the rapid 
products (e.g., librarian or information scientist, statistician, economist, project coordinator, 
researcher); often these staff also produced the standard systematic reviews or HTAs, or had 
experience with these more comprehensive products. Moreover, most often staff had extensive 
experience with reviews and graduate-level or medical training. The number of personnel 
assigned to a given product varied from zero (a computer algorithm where the user enters query) 
to nine; however, most common was one to three staff for shorter timeframes and two to four 
staff for longer timeframes. 

The target audience varied and included: hospitals, hospital management, health systems, 
health systems administrators, ministries of health, health plans, special interest organizations, 
societies, guideline development organizations, universities, medical manufacturers, industry, 
public health consultants, patients, clinicians, policy makers, and payers. However, health 
systems tended to dominate and few programs responded to requests from individuals not 
associated with a larger organization. 


All organizations described a structured topic nomination process that took different forms 
(e.g., dedicated phone line, email or electronic submission). Most organizations engaged directly 
with the nominator to refine the topic, adjust the scope as needed, and establish user needs and 
timelines. In fact, very few did not mention involvement with the nominator to refine the topic 
and develop the review. 

We analyzed this information in two different but complementary ways guided by the two 
aspects of the term “rapid review.” First, we summarize information on the rapid products 
according to their production time (i.e., how rapidly they are produced). Second, we divide rapid 
products into four categories based on the extent of synthesis—Evidence Inventories, Rapid 
Responses, Rapid Reviews, and Automated Approaches—and describe the methods used for 
each type of product. Appendix D provides four tables describing these products according to: 
(1) product scope and purpose, (2) comprehensiveness, (3) rigor and quality control, and (4) 
approach to analysis and conclusions. 


Description of Rapid Products by Production Time 


Table 2 summarizes the scope, end-users, and purpose of rapid products by production time 
(more detailed information available in Table E-1). The sections that follow provide a narrative 
summary of timeframe and methods (detailed information available in Tables E2-4). 


Table 2. Scope, end-user and purpose of rapid review products by timeframe 








(5 products) 


e 3 limited to emerging technology 

e 4 limited number of questions 

e 2 limited other aspects of scope 
(comparisons, questions, outcomes) 


payers, Clinicians, policy 
makers 


Timeframe Scope End-Users Purpose 
(number of 
products) 

Within 1 week | eAll products limited to single comparison Hospital administrators, | Give user a picture of 


available evidence and 
regulatory issues; 
coverage decisions; 
clinical decisions 





< 4 weeks 
(6 products) 


e All products limited to emerging technology 
e 2 limited to single comparison 

e 4 limited number of questions 

e None limited the number of studies 


Health system and 
hospital administrators, 
clinicians, payers, policy 
makers 


Allow user to do own 
research; coverage 
decisions, purchasing 
decisions; assess the 
need for future research 





4-8 weeks 
(11 products) 


(7 products) 


e 4 products limited to single comparison 
e 4 limited to emerging technology 

e5 limited number of questions 

e 2 limited the number of studies 

e 4 limited other aspects of scope 


e2 limited to emerging technology 

e 4 limited number of questions 

e2 limited number of studies 

e 1 limited other aspects of scope 
(comparisons, questions, outcomes) 


Policy makers, service 
providers, clinicians, 
hospital administrators, 
payers 


policy makers, research 
institutions, 
government, 
manufacturers, 
Clinicians, payers 


Purchasing decisions; 
designing care delivery; 
clinical decisions; 
coverage decisions; 
policy; allow user to do 


(comparisons, questions, outcomes) own research 
2-3 months ¢5 products limited to single comparison Hospital administrators, | Research prioritization; 


background; purchasing 
decisions; practice 
decisions; coverage 
decisions 





Minimum of 3 
months 
(7 products) 








e3 products limited to single comparison 

e 1 limited to emerging technology 

e5 limited number of questions 

e6 limited other aspects of scope 
(comparisons, questions, outcomes) 





Hospital and health 
system administrators, 
research institutions, 
government, industries, 
policy makers, payers 





Policy; coverage 
decisions; purchasing 
decisions 








Production Time: Within 1 Week 

Timeframe. Five products from five organizations were completed within 1 week (up to 5 days). 
These products were referred to as: ultra-rapid review, scoping review, search and summary, 
mini-review, and “‘answers.” Time to completion ranged from 5 minutes (computer algorithm in 
which user enters query) to “less than 1 week.” 


Methods. Typically, these products involved searching a limited number of electronic databases 
(1 to 3). The use of grey literature was generally limited; only two reported sources in addition to 
electronic databases. Most restricted the type of studies included; for example, one used existing 
reviews if possible, and another limited to controlled trials. Only one program reported that they 
reviewed the full text of articles. None of these products involved dual study selection or dual 
data extraction. Most focused on a limited number of extraction elements; two of them focused 
on only one outcome. Only one of the five products involved quality assessment and grading the 
quality of evidence. Most products involved some form of internal review (e.g., by director of 
the program), but none had external review (e.g., peer review). Most products provided a 
narrative review with minimal synthesis (i.e., integration of findings across studies). Two 
provided conclusions and/or recommendations; others allowed end-users to formulate 
conclusions. 


Production Time: Less Than 4 Weeks 

Timeframe. Six products from five organizations were completed in less than 4 weeks. Time to 
completion ranged from 5 to 10 business days to “less than 4 weeks.” These products were 
referred to variably, e.g., rapid response, brief, summary of abstracts, evidence inventory. 


Methods. Most indicated that the literature search, in terms of electronic databases, was limited; 
however, most listed at least four databases that were routinely searched. Most reported use of 
grey literature; only two reported that no grey literature searching was done. Five products 
reported on language restrictions and all of these focused on English only. Four products 
described limiting the search dates to the most recent 5 years, one indicated that date restrictions 
were sometimes considered, and one indicated no date restrictions. The use of primary vs. 
secondary sources varied: three products reported including systematic reviews and HTAs and 
one product did not include other designs if a systematic review or HTA was available. In four 
products, there was no full text review; in addition, one product only performed full text review 
if the abstract was not sufficiently detailed, and one product performed full text review but for 
limited information. All products limited the number of extraction elements and two products did 
no data extraction. None of these products involved dual study selection or dual data extraction. 
None of the six products involved quality assessment or grading the quality of evidence. Most 
products involved some form of internal review (e.g., by director of the program), but none had 
external review (e.g., peer review). Five of the products involved no synthesis, while one product 
included a qualitative synthesis. One product included key messages and another included 
opinions of the review authors with caveats. 


Production Time: 4 to 8 Weeks 

Timeframe. Eleven products from eight organizations were completed in 4 to 8 weeks. These 
products were referred to variably, e.g., technology brief, evidence brief, evidence summary, 
evidence advisory, summary with critical appraisal, rapid response, rapid review, and HTA. 


Methods. The extent of searching electronic databases varied considerably from 2 databases to 
“any and all” depending on the topic; the majority listed 4 to 5 databases. All conducted some 
searching of the grey literature. Five products described limiting to English language studies; 
only one product mentioned no language restrictions. Six mentioned possible limits to the search 
dates depending on the topic. Four products relied on existing systematic reviews or HTAs if 
available. Most indicated that they conducted full text review; in one case no data extraction was 
done. Five specifically mentioned that the number of extraction elements was limited. Only two 
reported dual study selection. One reported dual data extraction while two others reported 
extraction with verification by a second reviewer. Quality assessment was conducted in the 
majority of cases, with two specifically indicating no quality assessment. Five reported grading 
the quality of evidence, one reported that it had been done on occasion, and one reported that it 
was under consideration. Only two products described external peer review and in one case peer 
review was not always done. Where data were synthesized, a qualitative approach was used in all 
cases. None reported conducting a quantitative analysis. Five indicated that they provide 
conclusions and/or recommendations, two provide key messages, and four provide no 
conclusions or recommendations. 


Production Time: 2 to 3 Months 

Timeframe. Seven products from six organizations fell into this category where product 
completion averaged approximately 2 months, but ranged from 1-3 months to 2-4 months (Le., 
all had an upper limit of more than 2 months as distinct from the previous category). These 
products were referred to variably, e.g., literature summary, emerging technology report, “rapid 
and responsive HTA,” HTA, evidence review, and “snapshot” review. 


Methods. The extent of searching electronic databases was variable: two described searches as 
comprehensive, one limited searches to 3 electronic databases, and the other products listed 
examples of databases but did not provide detail on any limits. All but one product reported 
searching for grey literature. Of the four products that reported on language, all restricted to 
English only. Two indicated that searches were limited to the most recent 5 years, while two 
indicated that searches were sometimes limited by date. Only two products indicated that they 
rely on or preferentially use existing systematic reviews or HTAs. In five cases, reviewers 
conducted full text review, one did not, and one did not extract data. Additionally, three products 
reported that the number of extraction elements was limited. Only one product reported dual 
study selection, and no products reported dual data extraction. Three products did not do quality 
assessment, while two products did so in a limited fashion. Only two products reported grading 
the quality of evidence. Peer review was mentioned for only one product; however, three 
products mentioned external review by experts or manufacturers. Two products reported doing 
no synthesis, three reported a qualitative synthesis, and two reported conducting quantitative 
analyses when possible. Four products did not provide conclusions or recommendations, while 
three report conclusions but not recommendations. 


Production Time: Minimum of 3 Months 
Timeframe. Seven products from six organizations required a minimum of 3 months with 
maximum time to completion ranging from 4 to 8 months. These products were referred to 


variably, e.g., rapid review (in three cases), evidence check, streamlined review, and summary 
review. 


Methods. The extent of searching electronic databases varied considerably: limits were 
described in three cases, while in another three cases the searches were described as not limited 
or “comprehensive.” Most reported use of grey literature; only one reported that no grey 
literature searching was done and the other indicated that grey literature searching was decided 
upon with the nominator. Six products reported on language restrictions and all of these focused 
on English only. Four products indicated restricting the search dates, either to the most recent 5 
years or restrictions based on topic area and in discussion with nominator. In four cases, the 
product would rely on existing systematic reviews or HTAs if available; one of these was 
described as a review of reviews and would not include primary studies. In the majority of cases, 
reviewers conducted full text review. Four products limited the number of extraction elements; 
extent of data extraction was unclear for the remaining products. One product involved dual 
study selection, while methods for the remaining products were variable, e.g., sometimes a 
second person would check, or only check excluded studies. No products involved dual data 
extraction; four products reported single extraction with verification by another reviewer. Five 
products involved quality assessment, one indicated that quality assessment varies, and only one 
reported that no quality assessment was done. Four products indicated that grading of the quality 
of evidence was done, and one product indicated that this is at the discretion of the nominator. 
For four products, it was reported that external peer review was done. All products involved a 
qualitative synthesis; two specified that a quantitative synthesis is done if the data allow and one 
indicated that quantitative analysis is done at the discretion of the nominator. All products 
indicated that they provide conclusions and/or recommendations. 


A Typology of Rapid Review Products 


Once the data on different dimensions of the review process had been extracted, it became 
clear that while the methods that rapid review products use are very heterogeneous, they can also 
be fairly readily grouped into four categories according to the level of synthesis: evidence 
inventories, rapid responses, what we chose to call true rapid reviews, and automated 
approaches. 

e Evidence inventories list what evidence is available, and often other contextual 
information needed for making decisions, but do no synthesis and do not attempt to 
present summaries or conclusions. 

e Rapid responses organize and evaluate the literature to present the end-user with an 
answer based on the best available evidence but do not attempt to formally synthesize the 
evidence into a new conclusion. Usually this means reporting the conclusions of 
guidelines or systematic reviews, but some rapid response products apply a best evidence 
approach and report the results of primary studies if no secondary sources are available. 

e “True” rapid reviews perform a synthesis (qualitative, quantitative, or both) to provide 
the end-user with an answer about the direction of evidence and possibly the strength of 
the evidence. 

e Automated approaches are databases of extracted study elements that use computer 
algorithms to generate meta-analyses in response to questions. These are very different 
than other rapid products or systematic reviews, in that the search, extraction and grading 
are dissociated from the analysis, which is performed according to preset computer 
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programs. However, they do produce a synthesized conclusion in less than 6 months and 
therefore will be briefly described. 


Ten organizations produced only one type of product, six organizations produced two types 
of products, and three produced three types of products. Four produced more than one version of 
the same category of products, either to maximize what could be produced in the time allowed, 
or because they had more than one end-user with distinct needs. It should be noted that these 
categories are somewhat subjective, and that numbers do not always add up because it was not 
possible to get information on all dimensions for all products. Therefore, the numbers reported 
below give a sense of frequencies but may not be representative or reproducible. We have linked 
to examples primarily from the Canadian Agency for Drugs and Technologies in Health 
(CADTH) because CADTH produces three of the four types and all the reports are publically 
available; we also included one report from the Center for Evidence Based Practice to illustrate 
variation. 


Inventories 


General Description 

An example of an evidence inventory is the CADTH Literature Summary “Mobile Apps for 
Improving Health: Clinical Evidence.”!” This product presents a list of the publications available, 
organized by secondary sources, randomized controlled trials, and non-randomized controlled 
trials. The key message was “two systematic reviews, seven randomized controlled trials, and 
three non-randomized studies were identified regarding the use of nutritional and physical 
activity applications (apps) for promoting healthy lifestyles. No literature was identified 
regarding the use of nutritional and physical activity apps for preventing chronic disease.”!” This 
allows the reader to judge the extent of evidence and find it for themselves, but does not offer 
any answers about the direction of evidence. These products may be considered consultations 
and may assist decision makers in specifying the scope of their question(s). 

We identified six examples of inventories produced by six organizations, four based in the 
United States and two in Canada. Terms used included Search and Summary, Literature 
Summary, Evidence Inventory, Rapid Review, Hotline Response, Product Brief, Reference list, 
and QwikNote. The length of time needed to produce them ranged from 3 days to 6 weeks, with 
most taking around 15 working days. The most commonly cited report length was 10 pages. 
They were generally described as a list of available studies (with either abstracts or links to full 
studies) that addressed the questions, generally organized in some kind of best evidence 
approach. One also included summaries of guidelines or systematic reviews if available. A 
couple included contextual information relevant to implementation, such as regulatory approval. 
Six appear to be aimed at system-level decision makers (health system, payer, policy maker). 
One producer also specified clinicians as end-users, but in the context of providing library 
services to practicing clinicians who did not have access to a librarian in their practice setting. 
When a statement of purpose was available, it generally described providing a picture of the 
available evidence, and/ or providing the information to make an initial decision about coverage, 
purchase, or implementation, often to determine whether further research was needed. 
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Methods for Time Reduction 

Three products explicitly limited the scope of the report, either by limiting the number of 
comparisons and/or number of questions, or by limiting the product to emerging technologies 
with a small evidence base, but three did not mention limiting scope and none mentioned either 
explicit or implicit limits on the number of studies that could be included. Comprehensiveness 
was also generally not limited: three mentioned limiting the search to | to 2 databases, only one 
completely excluded grey literature while two put some limits. However, three inventories 
always set time limits on the literature search and one sometimes did. Three preferentially looked 
for existing systematic reviews or HTAs, one never included secondary studies, while all 
included primary studies. Only one undertook full text review and only three carried out even 
limited extraction. Rigor and quality control was tightly curtailed, however, with none carrying 
out dual selection or extraction, and only three undergoing internal review by a supervisor or 
internal expert. None underwent any external review. None synthesized the findings into a 
conclusion, beyond possibly whether there was enough evidence for a systematic review. 

In summary, evidence inventories appear to be a rapid product that is targeted at hospital 
administrators and payers and is designed to demonstrate what evidence is available to make 
decisions, including whether investing in further review is worthwhile. They are not necessarily 
limited by scope or comprehensiveness, but because they do not attempt any evaluation (beyond 
organizing studies by study type) or synthesis, they are on average the fastest type of product to 
produce. 


Rapid Responses 


General Description 

Two examples of rapid responses are the Center for Evidence Based Practice’s (CEP) 
Evidence Advisories and CADTH’s Summary of Abstracts. A CEP Evidence Advisory on 
administration of intravenous insulin reviewed and evaluated the available guidelines and 
concluded: “The guidelines suggest performing double-checks of insulin doses and insulin 
administration, but lack detailed information on personnel and practice. Neither of the guidelines 
provides references to specific studies supporting recommendations. [low-quality evidence].””° 
A CADTH Summary of Abstracts on Short-Acting Intramuscular Olanzapine for Acute 
Agitation concluded: “Three non-randomized studies were identified regarding the clinical 
effectiveness of short-acting intramuscular (SAIM) olanzapine for the treatment of acute 
agitation associated with bipolar disorder or schizophrenia. One study comprised only patients 
with schizophrenia or schizoaffective disorder. Compared with ziprasidone, patients taking 
olanzapine had a greater length of hospital stay. The authors reported that second generation 
SAIM antipsychotics were not associated with a shorter length of stay than treatment with 
haloperidol, required more injections than haloperidol, and were more costly.””” The report then 
goes on to describe two other studies. Both of these reports provide the reader with what the best 
available evidence suggests the direction of evidence is, but do not synthesize the body of 
evidence into an independent conclusion. 

We identified 10 examples of rapid responses produced by 9 organizations, 3 based in the 
United States, 2 in the UK, 3 in Canada, and | in Australia. Terms used included answers, 
evidence advisory, rapid response, tech note, and Responsive Innovation Evidence Review. 

The time taken to complete them ranged from 5 days to 3 months, with most time estimates 
falling around 1 to 2 months. Report length (where information was provided) ranged 
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from 1 to 15 pages, with under 10 pages being the most common. Of the products that targeted 
particular end-users, four were intended to be used by policy makers or health system 
administrators and one by payers for purchase or coverage decisions. Three were intended to be 
used by clinicians and one by service providers to support clinical care or design care delivery. 
(Note: some products had more than one targeted end-user). 


Methods for Time Reduction 

Scope was not necessarily a limiting factor—three specifically limited the scope, four did 
not, and three sometimes did. Some specifically mentioned that they were willing to consider 
broad and complicated questions. Only one was limited to new or emerging technologies. 
However seven limited the number of questions that could be asked and three either explicitly 
limited the number of studies that could be considered or else adjusted other parameters to 
ensure that the number of studies was not overwhelming. Of note, the rapid response that was 
produced in under 5 days was the most tightly constrained—limiting scope, number of questions, 
number of studies, and focusing on new technologies. 

The number of databases searched ranged from two to four and several reported a recursive 
approach, stopping the search as soon as enough data was acquired. Only two excluded grey 
literature, although two set limits. Five of the 10 rapid response products limited search dates, 
usually to the past 5 years, and a couple sometimes restricted inclusion to applicable settings. For 
the nine products for which this information was available, all preferentially used systematic 
reviews or guidelines; primary studies were only used if there were no reliable and recent 
systematic reviews. One product was only created if systematic reviews were available. For 
those for which data were available, about five did full text review, while three did not, but only 
limited extraction was done. There was no dual review or extraction (although there was 
oversight in one case). Four underwent internal review, one occasionally underwent internal 
review, and only one occasionally might go for external peer review—although some producers 
viewed public posting as a kind of peer review. 

Although most did not assess quality or risk of bias, two did and one sometimes evaluated 
the strength of the evidence. Three reported some level of qualitative synthesis (although without 
doing a quality assessment), and none attempted quantitative synthesis. The level of conclusion 
was generally presented as a summary of the best evidence, or a narrative discussion of the best 
evidence, with perhaps key messages or points to consider. 

In summary, as their name implies, rapid responses attempt to give a quick answer based on 
the best available evidence to support a purchase or clinical decision, and less often a coverage 
decision. There is more limitation of scope and comprehensiveness than with inventories. 
However, the key time saving is achieved by skipping synthesis and instead relying on existing 
systematic reviews or guidelines to provide a reasonably reliable answer. 


Rapid Reviews 


General Description 

An example of a rapid review is CADTH’s summary with critical appraisal “Bariatric 
Surgical Procedures for Obese and Morbidly Obese Patients.””' This report described and 
critiqued the available studies, and qualitatively synthesized the results to produce a conclusion 
that encompassed the totality of the evidence: 
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Results consistently demonstrated that Roux-en-Y gastric bypass (RYGB) was 
associated with a greater weight reduction relative to laparoscopic adjustable 
gastric banding (LAGB), but was also associated with a higher risk for 
procedural adverse events and a longer duration of hospitalization after the 
procedure. The evidence of effectiveness and safety for sleeve gastrectomy 
suggested that it is less effective than RYGB for weight loss but associated 
with a reduced risk for complications, and more effective for weight loss 
compared with LAGB, but also more likely to result in complications, but 
evidence was conflicting.”! 


This approach gives the reader an answer based on a new synthesis of the evidence and is 
most similar to a standard systematic review. 

We identified 18 examples of rapid reviews produced by 15 organizations, 6 based in the 
United States, 2 in the UK, 3 in Canada, 3 in Australia, and1 in Italy. Terms used included rapid 
reviews, evidence reviews, emerging technology report, streamlined review, summary review, 
evidence check, evidence summary, CompNote, Summary with critical appraisal, and Tech 
Brief. The time taken to complete them ranged from 3 days to 6 months, with most time 
estimates falling around 2 to 4 months. Report length (where information was provided) ranged 
from 1 to 150 pages, with 20 pages being the most common. Of those that defined an audience, 
seventeen were intended to be used by payers, policy makers, or health system administrators for 
coverage or purchase decisions, and only one was intended to be used by clinicians. 


Methods for Time Reduction 

Careful scoping appears to be the primary way most rapid reviews achieved their deadlines. 
Twelve limited the scope to a single comparison (another did sometimes), 8 limited questions to 
emerging technologies where it was known the evidence base would be sparse, 11 limited the 
number of questions that could be considered (1 did sometimes), and 15 limited the number of 
studies that could be included in the final report, either explicitly or through adjusting the scope. 
In some cases, a maximum number of studies were set and if the initial search went past the 
limit, the scope would have to be further reduced or the topic would have to be promoted to a 
standard systematic review. A few limited the number of database searches, but more said they 
used the same search pattern as for standard systematic reviews. Grey literature was excluded for 
only one, although four reported limited or variable use of grey literature. Nine always or 
sometimes set date limits on searches and a few used study size to limit inclusion. Ten used 
existing systematic reviews preferentially, but often mentioned updating them and only one 
never used primary studies. However five preferentially used primary trials, with existing 
systematic reviews used as reference sources or background if at all. Only one did not always do 
a full-text review, but almost all significantly restricted the number of items that were extracted. 
About a third had some kind of check on screening and extraction (although only one did full 
dual screening). Eleven had internal review, while 10 had some form of external review, either 
external experts or peer review. Sixteen did quality rating of the included studies, while 10 
formally rated the strength of evidence and 2 did sometimes or were considering doing so. All 
performed qualitative synthesis while seven also performed quantitative synthesis and some did 
both (six). All provided conclusions, although some added a number of caveats. 
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In summary, this type of report appears to be the most comparable to a standard systematic 
review, albeit one with a tightly constrained scope and that often sacrifices quality control 
measures in order to provide a timely response. 


Automated Approaches 

Two of the products that were identified in our search for rapid reviews did not fit neatly into 
the three categories described above. However, because they involved a systematic search and 
synthesis of the evidence to produce an answer about the direction and strength of evidence, we 
felt they met our broad inclusion criteria. Both products were essentially algorithms attached to 
search engines and databases of extracted studies that can be used to produce meta-analyses on 
command. Because we only had two examples, we were not comfortable making general 
observations about this type of rapid product. However, it is important to note that they do exist 
and represent a potentially attractive strategy to provide timely evidence for decision making and 
to support clinical care, given the labor intensive nature of most evidence reviews. Empiric 
research regarding the accuracy and reliability of these approaches is needed. 


Comparing Systematic Review and Rapid Review 
Approaches 


Philosophical Differences Distinguishing a Systematic Review and a 
Rapid Review 


Through our interviews with a large number of organizations who develop rapid products, it 
became clear that the development of a rapid product in most cases is not simply a “mini- 
systematic review with corners cut” but that the very essence of developing rapid products 
differs from that of systematic reviews. Table 3 describes thematic differences that arose from 
interviews. These core differences in large part explain the observed differences in products. 

One of the biggest differences between rapid products and standard systematic reviews is the 
relationship with the stakeholder. In many ways, this feature drives other differences. Rapid 
products are often conducted to help a specific end-user make a specific decision in an identified 
timeframe. Because of this, the question for the reviewer is often what can be provided in the 
time allowed. This in large part explains the broad portfolio of products often produced by rapid 
response groups (ranging from lists of relevant studies and guidelines to reports that include 
synthesis and evidence grading) and differences in formats compared with systematic reviews. 
The often compressed timeframe also explains the choice of some groups to rely heavily on prior 
systematic reviews and different presentation formats. Finally, maintaining a highly skilled staff 
is especially critical to rapid response groups who need to produce reports in a short timeframe 
and understand the type of products that might meet the needs of the decision maker. 
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Table 3. Philosophical approaches to rapid and systematic reviews (based on Key Informant 








interviews) 
Product Features Rapid Review Systematic Review 
Emphasis/Priority End-user. Provide information to Product: Conduct a comprehensive, 


help a specific decision maker make 
a decision 


unbiased and rigorous systematic 
review (often with multiple 
stakeholders in mind) 





Relationship With End-User 


Continuous intimate relationship with 
a specific end-user in iterative 
fashion throughout work to ensure 
product will meet end-user’s need 


Arms-length relationship with end- 
users engaged at specific time 
points, and review group separated 
from end-users during certain 
phases to reduce bias 





Role of Other Reviews 


Often high reliance on systematic 
reviews for information 


Often limited use of reviews. 
Systematic reviews one among 
many sources. The primary role of 
systematic reviews may be to 
identify primary studies 





Organizational Features/Staffing 


Spectrum of Products 


High reliance on maintaining highly 
trained staff to conduct reviews 


Broad range — feature determining 
type of product is time allowed and 
needs of decision maker 


Useful to have experienced staff but 
not essential (e.g., more 
time/possibility to train staff during 


conduct of the review 


Consistent comprehensive product 





Scope 








More routinely focused question 





Range from focused to broad 
questions 





Differences Between a Systematic Review and a Rapid Product 


Few rapid products encompass all the characteristics of a comprehensive systematic review. 
The following list highlights some key differences identified by the Key Informants (KIs) at the 
various stages of a systematic review. We compare these strategies to systematic reviews 
produced through the EPC Program. 





e Research question: The research question for a rapid product is generally narrower than a 


systematic review. Systematic reviews produced through the EPC Program often answer 
several broad overarching questions whereas rapid products commonly limit the number 
and scope of questions that they answer. KIs commonly mentioned focusing on new or 
emerging technologies (which have limited evidence) or single interventions or therapies 
within a narrow population. 

Topic refinement: Rapid response groups generally work closely with the nominator (or 
end-user) to refine the topic. The topic refinement process for a rapid product often 
involves a single reviewer meeting with an individual requestor, either via email or 
telephone. The scope and research question is often determined in a matter of days or 
weeks. Within the EPC Program, the topic refinement process for standard systematic 
reviews involves a team of reviewers working with a group of technical experts over an 
extended period of time, often up to several months. 

Literature search: The literature search for a systematic review is extensive, including 
multiple databases as well as grey literature searching, with a goal of capturing all 
available evidence on a given topic. Rapid responses often limit the search with very few 
organizations interviewed regularly searching more than five databases. Most 
organizations relied on Cochrane, PubMed and Medline and less than half of the rapid 
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products included a search for grey literature. Some organizations described stopping a 
search when “enough” evidence was found and using only existing systematic reviews 

e Literature Screening/Extraction: Systematic reviews often use a dual review process for 
full text article review and data extraction whereas most rapid response groups utilized a 
single reviewer and some did not include full text review. Most rapid products include 
data extraction performed by one person and some rapid products do not include any data 
extraction or severely restrict the number of data items extracted. 

e Synthesis: While it is common to perform quantitative analyses in systematic reviews, 
very few rapid products involved quantitative synthesis. Some rapid products included a 
qualitative analysis while some did not include any synthesis. 

e Review: Systematic reviews produced through the EPC Program often involve extensive 
internal and external review. Internal review follows a structured process with input from 
the Program and an arms-length editor from another EPC. External review involves 
public posting and traditional peer review. Some rapid products included an internal 
review and few included external review. 

e Overall process: Overall systematic reviews are more comprehensive in their approach; 
those conducted through the EPC Program follow established methods and procedures. 
Rapid products have greater variation in methodological practices in order to achieve a 
product in a shortened time span. 


Potential Concerns in Conducting Rapid Reviews 


Several of the interviewed organizations expressed concerns in developing rapid products. 
Due to the shortened timeline of the products, the process often involves a narrow and/or a 
limited selection and review of the literature. This process increases the risk of missing evidence. 
Additionally, the shortened timeline under which to review and consider the available evidence 
limits the thought process. This can trigger inherent uncertainty in the findings of a rapid 
product. Another concern expressed by rapid product producers is that the final products may be 
mistaken as full systematic reviews by certain requestors (i.e., reviews that have followed 
accepted standards for systematic reviews). This may especially be a concern for organizations 
that produce both full systematic reviews and rapid products. Explicitly outlining the trade-offs 
and limitations of a rapid product as compared with a systematic review at the beginning of the 
process can help to eliminate the chances of the rapid product being interpreted as a 
comprehensive review. Further, some organizations place disclaimers on the report in an effort to 
avoid any misinterpretation of what the product offers. 


Empiric Evidence Comparing Results of Rapid Products With 
Systematic Reviews 


Electronic and hand searches identified 12 studies that appeared to represent empiric 
evidence comparing results of rapid products with systematic reviews but only two studies 
presented empiric evidence related to this current project.'4???° Both studies were conducted in 
the context of HTAs. 

Van de Velde et al.”° presented a case study comparing results of a rapid review on potato 
peels for burns with those of a systematic review. The authors reported that the rapid review 
identified three RCTs, which was two more than were included in the systematic review. The 
results and conclusions of the two reports were different. However, the limited detail on the 
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methods used to conduct the systematic review makes this case study of little value for our 
purposes. 

In 2007, the Australian Safety and Efficacy Register of New Interventional Procedures — 
Surgical (ASERNIP-S)'* examined the use of rapid reviews in HTA. As part of the report, the 
authors compared the results and conclusions of rapid reviews with those of full systematic 
reviews or HTA reports. They searched websites of HTA agencies to identify rapid reviews on 
surgical interventions published between 2001 and 2004. Rapid reviews were defined as “any 
HTA report or systematic review that took between 1 and 6 months to produce and included a 
comprehensive or systematic literature search.” The rapid reviews were then matched with a full 
systematic review (i.e., production was not limited by time), written on the same topic, and 
produced within | year of the matching rapid review. 

Seven rapid reviews with matching full reviews were identified. The reviews addressed four 
topics: drug eluting stents, lung volume reduction surgery, living donor liver transplantation, and 
hip resurfacing. For living donor liver transplantation, there was one rapid review; for the 
remaining topics there were two rapid reviews each. The authors presented the clinical outcomes 
(safety and efficacy) discussed by each review, as well as the overall conclusions, the scope of 
the report, and the methods used. 

Differences in methods were identified between rapid and full reviews. The methods and 
results of full reviews were reported in more detail than the rapid reviews. Some rapid reviews 
included only systematic reviews or HTAs, while others conducted searches for and included 
primary studies. Most rapid reviews were written by single authors or authorship was not 
reported. Most rapid and full reviews underwent an external review. Searching was not 
consistently more extensive in full reviews (based on number of databases searched). Full 
reviews tended to assess the quality of primary studies, while rapid reviews either did not or did 
not report. Despite these differences, there were no instances in which the essential conclusions 
of the rapid and the full reviews were opposed. The full reviews consistently provided greater 
depth of information and more detailed recommendations pertaining to the implementation of 
each surgical intervention. The authors commented that full reviews may be more appropriate to 
identify safety outcomes and risk factors associated with an intervention. 

The comparisons between rapid and full reviews are limited to the context of HTAs, which 
are often driven by need for more rapid advice on explicit decisions about use of new health 
technologies, many with very limited published evidence. Due to the context and the relatively 
small volume of research, the HTA-based assessments may be only modestly applicable to 
broader topical systematic reviews with much larger bodies of evidence. 
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Discussion 


Our review of the literature and interviews with prominent producers of rapid products 
confirms the findings of previous reviews that there is a range of products that are being used to 
support evidence based decisions that can be produced more quickly than standard systematic 
reviews. Further, the terms used to describe rapid products and the methods they employ vary 
widely. 

Our analysis of these products by production time generally showed that increasing amounts 
of time allowed incorporation of more of the standard systematic review methods and quality 
control mechanisms. For example, the most rapid products (within 1 week) generally: searched a 
limited number of databases; limited use of grey literature; restricted the types of studies 
included; did not conduct full text review; did not perform dual study selection or data 
extraction; focused on a limited number of extraction elements (e.g., only one outcome); did not 
do quality assessment or grading; provided a narrative review at most with minimal synthesis; 
and provided minimal conclusions and/or recommendations. Further, none of the most rapid 
products underwent external peer review. As the timeframes increased many of these limitations 
were lifted, e.g., more databases searched, use of grey literature and full text review. However, 
even among the products that took longer to complete (minimum 3 months and up to 8 months), 
there were variable limitations on database searching, language (i.e., using English only), and 
search date (e.g., using the most recent 5 years). Further, some relied on existing systematic 
reviews or HTA reports if available. While there was an increase in full text review with 
increased production time, the number of extraction elements was still generally limited. In 
addition, use of dual study selection was variable; most did single data extraction, with data 
verification by another reviewer at most. Among the reviews with longer production timeframes, 
there was an increase in the frequency of quality assessment, evidence grading, and external peer 
review. 

To gather a better understanding of the variation, we categorized products based on the 
extent of synthesis performed, and found that we could describe four distinct types of products, 
with slightly different approaches and audiences. 

e Inventories list what evidence is available, and often other contextual information needed 
for making decisions, but do not synthesize the evidence or present summaries or 
conclusions. They are generally systematic and comprehensive; therefore, can provide a 
useful first step for decisions about coverage or purchase, in particular whether further 
research is warranted. 

e Rapid responses evaluate the literature to present the end-user with an answer based on 
the best available evidence (usually guidelines or systematic reviews), but do not attempt 
to formally synthesize the evidence into a conclusion. This type of product was more 
likely to describe clinicians among end-users. 

e “True” rapid reviews perform a synthesis (qualitative, quantitative, or both) to provide 
the end-user with a conclusive answer about the direction of evidence and possibly the 
strength of the evidence. The conclusion may be hedged, but it is still a synthesized 
statement that could be compared with a systematic review. They are generally able to 
meet time constraints by limiting the scope, are somewhat less comprehensive in their 
data sources, use fewer quality controls, and the synthesis is usually qualitative. 

e Finally, automated approaches are a new and intriguing approach to providing rapid 
evidence support, and worth future study. 
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Moving Beyond Methods 


Through our interviews with a large number of organizations who develop rapid products it 
became clear that the conduct of a rapid product in most cases is not simply a “mini-systematic 
review with corners cut” but that the very essence of developing rapid products differs from that 
of systematic reviews. One of the biggest differences between rapid products and traditional 
systematic reviews is the relationship with the stakeholder. Rapid products are often conducted 
to help a specific end-user make a specific decision in an identified timeframe; therefore, the 
reviewers needs to make decisions about what they can provide in the time allowed. This in large 
part explains the broad portfolio of rapid products and reporting formats. The timeframe also 
explains the choice of some groups to rely heavily on prior systematic reviews. Maintaining a 
highly skilled staff is critical to rapid response groups who need products in a short timeframe 
and understand the type of products that might meet the needs of the decision maker. Finally, 
few and/or narrow questions were also a necessary approach, focusing on new or emerging 
technologies or single interventions within a narrow population. 

Based on our review of the literature, there is almost no empiric evidence comparing results 
of rapid products with systematic reviews. For the most part, the research represents indirect 
evidence about whether truncating methods have an impact on results and conclusions. One 
report that provided direct evidence suggests that there may not be any impact; however, this 
report was in the context of HTAs and focused exclusively on surgical interventions often early 
in their implementation. The systematic and rigorous approach implied in the use of the term 
“review” cannot clearly be completely fulfilled in a more rapid process, but little is known about 
the impact of truncated methods. Thus, shorter may not always be better, or even as good, but 
currently available information does not tell us if and when to be concerned. For example, 
previous research has suggested that lack of quality or risk of bias assessment may result in over- 
representation of or inappropriate emphasis on poor quality research.'’ Further, previous research 
has raised the concern that rapid products may not comprehensively address safety issues.'* The 
issue of safety assessment did not specifically arise during the Key Informant interviews; 
however, we found that often rapid products were limited in the scope of the question(s) and the 
number of outcomes examined, often with a focus on efficacy outcomes. Closer examination of 
the appropriateness of rapid products for issues of safety is needed. Finally, many rapid products 
rely on qualitative synthesis which can be more challenging than quantitative synthesis in terms 
of determining effectiveness and assessing heterogeneity; moreover, many approaches to 
qualitative synthesis rely on vote counting methods which are not recommended within 
systematic review standards. 

In summary, rapid products have tremendous variation in methodological practices; however, 
what appears to tie them together is the close relationship with the end-user to generate an 
evidence base that meets their decision-making needs within a limited amount of time. To date, 
there is little empiric evidence regarding the validity of results and conclusions based on rapid 
approaches to reviewing the literature and this was an expressed concern of the interviewees. 


What Was Known and What This Paper Adds 


Our findings are consistent with previous reviews showing that rapid review methods vary 
greatly, as do their definitions and applications. For example, a 2010 systematic review of “rapid 
reviews” examined 45 methodological articles and 25 exemplars of rapid review methods and 
found many subtle differences among terms used to denote a more accelerated production 
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process (e.g., rapid, ultra rapid, succinctly timed) for a type of product (e.g., technology 
assessment, systematic review, evidence assessment). These differences were magnified when 
considering the associated timeframes which, although not consistently reported, ranged from 
one to nine months, and overlapped in the time required for or even as long as some traditional 
systematic reviews and more comprehensive evidence synthesis products (e.g., technology 
assessments). Also consistent with our findings, the earlier systematic review found that rapid 
reviews varied in their methodological approaches and how closely these adhered to current 
standards for systematic reviews.'| We showed that this methodological variation was related to 
production time with more standard methods incorporated where more time was available. As we 
did not review the actual rapid products, we did not assess quality of reporting. Previous reviews 
and overviews on rapid reviews also raise concerns about potential impacts on the validity of 
information resulting from truncated methods or timelines.” '!* Our search for and review of 
empirical evidence showed a substantial gap in this area. 

The 12 previous reviews we identified that examined different rapid products do not support 
substituting any form of rapid product for a standard systematic review, although investigators 
recognize the value of rapid products for answering narrow efficacy or effectiveness questions 
within a shorter timeframe and with fewer resources. Our findings showed that often the rapid 
products are geared towards narrow questions, and that producers take time to appropriately 
scope the topic in order to meet the constrained timeframes. Previous reviews document methods 
for shortening the length to complete rapid products, including searching fewer sources, 
excluding grey literature, applying search filters, and employing only one reviewer for 
title/abstract reviewing or data extraction. These reviews also found that meta-analyses were 
rarely conducted in rapid products, although recommendations for producers of rapid products 
include meta-analyses as worthwhile additions to these evidence summaries. Previous reviewers 
noted that caution is needed when interpreting evidence when quality assessment has not been 
done, and recommended enhanced transparency in reporting methods used for the rapid product. 

Our data collection and analysis build on the findings of this previous work in three 
important ways. First, as described above, we recognized early on that there was substantial 
variability across the rapid products and that “rapid reviews” does not reflect a specific 
methodological approach. As such, we sought to classify these products in a meaningful way. 
We found a natural grouping according to the extent of synthesis done as part of the report and 
classified accordingly as inventories (no synthesis), rapid responses (some organization and 
evaluation of the literature), and rapid reviews (synthesis—qualitative, quantitative, or both). 
Second, we analyzed the methods by production time showing not simply variation in 
approaches but instead a systematic increase in comprehensiveness and incorporation of more 
standard review methods with increased production time. Third, and our unique contribution, 
was the interviews with producers of rapid products and identification of items beyond the 
technical aspects of producing reports that seemed to underpin the essence of rapid products. The 
key factors were the relationship with the end-user and the focus on producing evidence to meet 
a very specific decisionmaking need. In this context, rapid products may be a reasonable 
substitute for standard systematic reviews, so long as the end-user understands any potential 
caveats arising from the methods employed. Rapid products may be a reasonable alternative to 
standard systematic reviews in the following cases: urgent decisions, decisions not likely to have 
negative health consequences for large populations, decisions that can be easily changed if new 
information leads to a change in the appraisal, decisions about low risk interventions, and 
decisions about low cost interventions. The utility and appropriateness of rapid methods for 
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safety outcomes needs further evaluation. A further consideration is opportunity costs, e.g., 
having rapid products on multiple questions of interest to the end-user versus standard systematic 
reviews for just one or two. We also identified key aspects of the organizations that allow them 
to produce rapid products, including the capacity to respond quickly to end-user requests and 
mobilize a highly trained and experienced staff. Our Key Informant interviews reinforced the 
findings of previous studies highlighting the need for health care researchers to focus on the 
informational needs and preferences of their target audiences.” This focus ensures the conduct of 
relevant and timely research that decision makers can directly apply to their local contexts.”>° 


Pragmatic Considerations 


Based on our review of the literature and interviews with Key Informants with extensive 
experience developing rapid products, we offer the following items that should be considered by 
a group or organization prior to implementing a program that offers reports in a limited 
timeframe. We believe that many of these factors need to be in place for an effective rapid 
response program. 

e Context: Many organizations reported producing reports for very specific decisionmaking 
needs in very specific circumstances. The evidence needed to inform decisions in this 
context is quite different from a systematic review that is intended to be used by multiple 
stakeholders (who are not necessarily all known or identified during preparation of the 
review). 

e Relationship with the end-user: Most organizations described a continuous and close 
relationship with a specific end-user involving iterative feedback throughout the work to 
ensure the product would meet the end-user’s needs. This is in contrast to a more “‘arms- 
length” relationship with end-users who may be engaged at specific time points during a 
systematic review, but who may also be separated during certain phases to reduce bias. 

e Reliance on secondary sources: Many organizations relied on existing systematic reviews 
or HTA reports. Methods need to be in place, and acceptable to the end-users, to 
synthesize and interpret evidence from existing reviews. 

e Capacity: Most organizations had extensive experience producing rapid products and had 
a highly skilled staff that either also prepared standard systematic reviews or HTA 
reports, or had experience doing so. The ability to mobilize skilled staff quickly was 
critical for a timely response, efficient back-and-forth communications to refine the 
questions and scope, and an expedited product. 

e Scope: Many organizations restricted the scope of the review to meet reduced timelines. 
Often questions focused on single interventions, single comparisons, specific populations, 
or specific outcomes. Many focused on new or emerging technologies for which little 
evidence was available resulting in the ability to review evidence quickly. 

e Methods: The reality of rapid products is that they are not necessarily as comprehensive 
as systematic reviews and do not meet all the accepted methodological standards. The 
producers and users need to be able to accept some modification to the methods in order 
to meet constrained timelines. Interestingly, some interviewees commented that 
individuals who came from a systematic review background often had a harder time 
making methodological concessions in the interests of efficiency. 

e Reporting and caveats: Pragmatic decisions to limit a review will lead to a tradeoff 
between preparation of a review that is timely and useful with potentially less accurate 
findings, and one that is comprehensive and as accurate as possible.”’ Central to the rigor 
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of the systematic review approach is transparency. As with any systematic review, the 
methodological approaches and accompanying justifications need to be clearly reported 
and also reflected on in terms of potential bias and shortcomings in the review 
conclusions, as well as the potential value of a more comprehensive review.”” Finally, the 
issue of protocols and pre-specified methods was not specifically addressed in this work 
but warrants consideration in the context of rapid products in the interest of transparency. 


Strengths and Limitations 


Our combined approach of reviewing the literature and conducting Key Informant interviews 
provided a rich source of information on rapid products, methods, and organizational 
perspectives. We analyzed and classified rapid products in unique ways that complement 
existing literature. We identified a select set of Key Informants from a number of well-known 
organizations that produce rapid products; however, the results may not be representative of all 
organizations or groups that produce rapid products. We also identified products through our 
literature review; however, programs captured in this manner may or may not be currently active. 
We sought to be comprehensive and consistent in the information gathered through the 
interviews and literature in order to accurately describe all rapid products; however, sometimes 
the information was limited in the literature and not all interviewees responded to our queries to 
confirm details. We gathered information on the overall timeframes for completion of rapid 
review products; however, we do not have detailed information on the total work hours involved 
in preparing different types of reviews. Further, we do not know whether the timeframes for 
production reflect real or artificial timelines (i.e., a report is produced in 2 weeks but is there an 
actual decision waiting to be made at the end of those 2 weeks). We found very limited data on 
the validity of rapid products and this should be a priority for future research. Of specific interest 
is the validity of rapid products for assessing safety outcomes. A point that arose while drafting 
the report is the issue of durability of the findings of rapid products. The issue of updating for 
standard systematic reviews has received much attention among methodologists and producers of 
systematic reviews and warrants consideration within the field of rapid products. 


Future Directions 


The work reported in this paper has helped us understand the approaches undertaken in the 
production of rapid products, and perspectives from relevant organizations. Many of these 
organizations have well-established rapid response programs that have existed for years. This 
fact underscores the value and role of rapid products in some situations and for particular end- 
users. A next important step is to assess the end-user perspectives on different rapid products, 
and the acceptability of different approaches to support their decision-making needs. To this end, 
we plan to conduct additional work gathering feedback from different end-users on concrete 
rapid product scenarios. These will help highlight the most important differences and trade-offs 
between rapid products and standard systematic reviews, and help guide decision making about 
when and where particular products would be helpful. Our results showing variation in rapid 
products by both timeframe and extent of synthesis will help us establish an interview guide to 
assess the relative importance of these factors in meeting end-user needs. Further, we will be 
able to ascertain how closely the timeframes and products are tied to actual decision making 
(e.g., 1s a report produced in 2 weeks used to make a decision at the end of those 2 weeks). 
Finally, we plan to address the issue of updating or “durability” of the products and whether this 
is an issue or concern for end-users. 
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We searched for evidence that compared the results and conclusions of rapid products with 
those from standard systematic reviews, and found little empirical evidence. We did not 
specifically search for empirical evidence on different aspects and/or methods that may be 
modified to conduct a review more rapidly (e.g., restricted number of databases searched, 
recursive approach to searching, restrictions on study designs eligible for inclusion, limited data 
extraction, lack of quality or risk of bias assessment, reduced quality control mechanism [1.e., 
dual study selection and data extraction], analytic approaches). However, this paper provides a 
detailed examination of the granular differences between products that are generated within 
different timeframes and offers a structure for detailed empirical investigations of how each of 
these methods impact final results and conclusions, and inform decisionmaking. 

Finally, given that most reviews have found heterogeneity across rapid products, it might be 
helpful in the future to apply functional groupings such as the typology used in this paper 
(Inventory, Response, Review, and Algorithm). In addition to allowing more conclusive analysis, 
this kind of framework could also be useful for evolving best practices—since best practices will 
clearly need to differ based on level of synthesis—and also for any empirical work either 
comparing rapid products with systematic reviews, or testing the validity of rapid methods. 


24 


Conclusions 


There is extensive variability in products commonly labelled “rapid reviews.” These products 
vary on two important dimensions which are captured by the term “rapid review:” the timeframe 
for completion and extent of review or synthesis. Our review of these products resulted in a 
classification according to the level of synthesis that we feel is a useful contribution to this field: 
inventories (no synthesis), rapid responses (some organization and evaluation of the literature), 
rapid reviews (synthesis—qualitative, quantitative, or both), and automated approaches 
(computer-generated responses to user-defined queries). We found a systematic increase in 
comprehensiveness and incorporation of more standard review methods with increased 
production time. There are a number of dimensions beyond the technical aspects of producing 
reviews that seemed to underpin the essence of rapid products, including the relationship with 
the end-user and the focus on producing evidence to meet a specific decision-making need. The 
well-established and longstanding programs among organizations that produce rapid products 
underscore the value of these products for end-users as a basis for decision making. More in- 
depth evaluation of end-user perspectives on rapid products will complement our findings and 
together will provide a more complete set of considerations for those interested in establishing a 
rapid response program or producing rapid reviews. 
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Appendix A. Search Strategy 


32O0vid MEDLINE(R) 1946 to October Week 5 2013 

Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations November 6, 2013 

Searched: November 6, 2013 

fe ((rapid or mini or pragmatic or targeted or focused or brief) adj2 (((systematic or evidence or data or 
knowledge) adj2 (review* or synthes*)) or HTA or health technology assessment”)).ti,ab. 


ie ((rapid or pragmatic) adj2 (review* or HTA or health technology assessment* or evidence 


assessment’)).ti,ab. 


1or2 
3 not animals/ 


remove duplicates from 4 
6 limit 5 to (english language and yr=“2000 -Current”) 





Ovid EBM Reviews - Cochrane Methodology Register 3rd Quarter 2012 
Searched: November 7, 2013 


((rapid or mini or pragmatic or targeted or focused or brief) adj2 (((systematic or evidence or data or 
knowledge) adj2 (review* or synthes*)) or HTA or health technology assessment”)).ti,ab. 





2 |(rapid adj2 review’).ti,ab. 
3 


SRC Methods Library 
Searched: October 21, 2013 
Descriptor search: systematic reviews — rapid 


27 records retrieved 
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Appendix B. Interview Guide 


Introduction 


The overall mission of the Agency for Healthcare Research and Quality’s (AHRQ) Effective 
Health Care (EHC) Program is to provide evidence-based information to health care stakeholders 
that is relevant to their needs, timely, objective, scientifically rigorous in construct, and 
developed and presented with transparency. In the production of systematic reviews, we aim to 
answer questions about effectiveness of interventions and average population effects. We are 
aware that for certain conditions and behavioral interventions, these questions may miss 
important issues. 

AHRQ engages stakeholders in all facets of their research enterprise, including the producing 
of systematic reviews, with the goals of ensuring that research findings reflect the needs of 
diverse users, are relevant to their unique challenges, and are applicable in real-world situations. 


Purpose of the Stakeholder Interview 


The goal of our project is to understand methods you have used, challenges you and others 
frequently encounter, and your advice on producing rapid reviews. 

We are very interested in learning from your experience. 

There are not right or wrong answers, so please feel free to share your thoughts openly. 

We would welcome any materials that you would like to share with us either before or after 
the interview session. Please send any materials to Johanna.anderson2 @va.gov. 


Ground Rules for the Stakeholder Interview 


The interview will be tape recorded, transcribed, and analyzed for overarching themes. 

Although the report may list individuals who were interviewed, answers will not be 
identifiable to individuals or specific organizations. 

You may refrain from answering any questions and are welcome to end the interview at any 
time. 


Interview Guide 


1. How would you define a rapid review (in regards to time, scope, etc.)? 
2. How long have you been doing rapid reviews? 
3. Who do you produce rapid reviews for? 
What is your organizational structure for how you produce rapid reviews (i.e., staffing, 
organizational capacity)? 

a. Who does your rapid reviews? What types of skills/expertise do they have? 
How do you determine which instances require a rapid review? 
What is a typical timeline and scope? Representative timeline and scope? 
Can you describe the methods used to produce a rapid review for each part of the review 
process (scoping, searching, extraction, synthesis, peer review, conflict of interest)? 

a. In what ways do you gain efficiency in the rapid review process? 

b. Are there any technologies or other tools that help you gain efficiency in the 

process? 


DO 
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7. Do you have any formal guidance documents that you use internally? Have you 
published on your methods? 

8. What do your rapid review products look like (how long do they take, how many pages, 
etc.)? 

9. What would you say are the advantages of the methods you used to make the process a 
rapid review? What are the disadvantages, if any? 

10. What has been your experience with end-users and how they accept rapid review 
methods? 

11. Would you be willing to share any examples of your work? 
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Appendix C. Flow of Studies from the Literature 
Review 


Figure 1. Search flow diagram 
Search results after duplicates 
removed: 468 


116 Included 











352 Excluded 


27 articles 
from 143 full-text 


workgroup articles 
and key reviewed 


90 Excluded 


53 included 
articles 


informants 


Background: Empiric Review of Rapid review 
8 data: 5 rapid review methods: 30 
types: 12 
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Appendix D. Characteristics of Rapid Products 


This appendix provides four summary tables of rapid products according to: (1) scope and purpose, (2) comprehensiveness, (3) 


methodological rigor and quality control, and (4) approach to analysis, synthesis, and conclusions. There are 36 products produced by 
20 organizations; these are ordered by classification and timeframe. 


Table D-1. Scope, end-user, and purpose of different rapid review products 






























































Scope 
Classification Timeframe emit te a End-user Purpose 
Limit to single emerging Limit number | Limit number 
comparison technology of questions of studies 
E1 Inventory 3-5 days F Give user a picture of 
Y Y Y N ey available evidence 
y and regulatory issues 
Al Inventory 5-10 days Health system Allow user to do own 
N N U N f ae, 
admin, clinicians research 
L1 Inventory 5-15 days Hosp Admin, Payers-coverage, 
. Hospital admin for 
Y N Y N Payers, Policy h fF 
fiakere purchase, policy 
makers for both 
R1 Inventory 1-2 weeks Health system shee need Wy 
Y N Y N administrators uibreeseaten ln 
alic kere support of purchase 
paey and coverage 
i Inventory <4 weeks Hospital admin, Assess need for 
N N N N Hate 
clinicians future research 
G1 Inventory 1-2 months N N N N Nain Research 
prioritization 
C1 Rapid response 3-5 days Y Y Y [Y] Payer Coverage 
D1 Rapid response <1 week 
(could be <1 Y N Y N Clinician Clinical decisions 
day) 
A2 Rapid response 15 days Variable N Y N Policy maker U 
D3 Rapid response 2-4 weeks N N Y N Policy maker Coverage 
FA Rapid response 4 weeks N N Y N Policy maker Purchase 
U1 Rapid response 1 month Service Designing care 
07; N U U U : : 
(minimum) providers delivery 
71 Rapid response 2-6 weeks N N N N Clinicians Clinical decisions 
12 Rapid response 4-8 weeks : Hosp admin, Purchase, clinical 
vatable nN = i clinicians decisions 
R2 Rapid response 1-3 months Y N Y Variable Policy makers U 
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Table D-1. Scope, end-user, and purpose of different rapid review products (continued) 




























































































Rapid response 10-12 weeks Research inst, 
Variable N U U govt, Background 
manufacturers 
Rapid review 3 days Y Y N [Y] Payer coverage 
Rapid review 4 weeks N Y Y Y] Hospital admin, coverage 
payer 
Rapid review 1 month Y Y N [Y] Hospital Admin purchase 
Rapid review 30 days Y N Y [Y] Policy, admin U 
Rapid review 30 days Y N Y [Y] Policy, admin U 
Rapid review 4-6 weeks (up N Y N N Policy maker, Purchase, policy 
to 16 max) hosp admin 
Rapid review Rae ae (up Y Y N Y Hosp admin Purchase/deployment 
Hala review at Y Y U U Hospital admin purchase 
Rapid review 8-12 weeks Hospital admin, Purchase, practice 
. i [Y] clinicians decision 
Rapid review 2-4 months Y N Y Y Payer coverage 
Rep ityevlow a ous Y NG Y Y Real ee Coverage, purchase 
Rapid review 3-4 months N N Y Y] dae ial U 
Rapid review 3-4 months Variable N U U pera tat pene ade 
Rapid review 3-4 months N N Y Y] Policy maker, U 
hosp admin 
Rapid review 3-6 months Y N Variable [Y] U U 
Rapid review 3-8 months Y N Y Y Payer coverage 
aaplareview oe Oninis Y N Y [Y] Payer coverage 
ma pIGeview eas N Y Y [Y] fie! Laat Purchase, coverage 
Algorithm 5 minutes y N Y N ‘ saa U 
olicy makers 
Algorithm 1-6 weeks N N N Around 30 U Allow maonean own 





* - working code used for internal purposes 
[Y]: No set limit for studies, but other aspects of scope (comparisons, questions, outcomes) adjusted to reduce the number of studies to a manageable number for the timeframe. 


Note: If this aspect was described as sometimes limited, this was counted as a “Yes” 


U = unknown 
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Table D-2. Comprehensiveness of different rapid review products 





























































































































bateey Grey Language Date Other search Use Existing SRs eke 
Classification TES electronic eee restrictions | restrictions | restrictionst and/or HTAs Eee 
databases 
E1 Inventory 3-5 days 2 Yes English Sometimes _| Sometimes No Yes 
Al Inventory 5-10 days 1+ Limited Yes Yes Yes Yes Yes 
L1 Inventory 5-15 days 1+ Yes Yes Yes No Yes Yes 
R1 Inventory 1-2 weeks U Limited Yes Yes U U Yes 
11 Inventory <4 weeks 4+ Yes Yes Sometimes _| Sometimes Yes [Yes] 
G1 Inventory 1-2 months U U U U U U U 
C1 Rapid response 3-5 days 3 U U U U U U 
D1 Rapid response <1 week (could 1+ No U No U Yes [Yes] 
be <1 day) 
A2 Rapid response 15 days 1+ Limited Yes Yes Yes Yes [Yes] 
D3 _| Rapid response 2-4 weeks Up to 4 No U No No Yes [Yes] 
FA Rapid response 4 weeks 2 Yes Yes U U Yes No 
U1 Rapid response 1 month 3+ Yes Yes Sometimes | No Yes Yes 
(minimum) 
T1 Rapid response 2-6 weeks 4 Yes U U Sometimes Yes Yes 
12 Rapid response 4-8 weeks 4 Yes Yes Sometimes _| Sometimes Yes No 
R2__| Rapid response 1-3 months 2+ Limited U Yes U Yes Yes 
O1 Rapid response 10-12 weeks 3 Yes Yes Yes U Yes [Yes] 
Bi Rapid review 3 days 2 Yes U U U U Yes 
E2__| Rapid review 4 weeks 2 Yes U Sometimes | No No Yes 
H1 Rapid review 1 month 5 U Yes Yes U Yes [Yes] 
A3 Rapid review 30 days 1+ Limited Yes Yes U Yes [Yes] 
A4_| Rapid review 30 days 4 Limited Yes Yes U Yes [Yes] 
J1 Rapid review 4-6 weeks (up to | 2-3 U U U U Yes [Yes] 
16) 
C2 Rapid review 4-8 weeks (up to | 4 Yes U Sometimes | U Yes Yes 
10) 
$1 Rapid review 2 person-months | U U U U U U Yes 
13 Rapid review 8-12 weeks 4+ Yes Yes Sometimes | Sometimes Yes [Yes] 
L2 Rapid review 2-4 months 4 Yes Yes Sometimes | Yes [No] Yes 
L3 Rapid review 2-4 months 5 Yes Yes U Sometimes [No] Yes 
G2__| Rapid review 3-4 months U Yes Yes No No Yes [Yes] 
O2__| Rapid review 3-4 months 3 Yes Yes Yes No [No] Yes 
P41 Rapid review 3-4 months Variable Sometimes | Yes Sometimes | U Yes Yes 
R3__| Rapid review 3-6 months 2+ Limited U Yes U Yes Yes 
M1 Rapid review 3-8 monthst 2 Yes Yes U Yes Ye [Yes] 
M2 | Rapid review 3-6 months (NS) | 2 Yes Yes U Yes Yes No 
N1 Rapid review 3-6 months (NS) _| U No Yes Sometimes | Yes Yes Yes 
D2__| Algorithm 5 minutes 1 No U No No No Yes 
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Number Grey Use 
‘ peta ; of : Language Date Other search Use Existing SRs : 
Classification WS electronic dbl restrictions | restrictions | restrictionst and/or HTAs baie 
databases | Searches studies 
Ki Algorithm 1-6 weeks 1+ Yes No No No [No] Yes 








* - working code used for internal purposes 
+ - these additional restrictions could include limiting studies based on geographic region, setting of care, minimum number of patients, duration 


£ - small (3 months), medium (6 months), large (8 months) 


[Yes] — only use primary studies if no SRs or HTAs 


[No] — not the focus 
NS = not specified; interviewee did not provide more detail on timeframe 


U = unknown 
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Table D-3. Methodological rigor and quality control of different rapid review products 





























































































































Limited data 
. ae : Full text Dual stud Dual data Internal External 
GEES ites et Hinetrane review elements Ssiscuont extraction review review 
extracted 

E1 Inventory 3-5 days No Yes No No No No 
Al Inventory 5-10 days No N/A No N/A Yes No 
L1 Inventory 5-15 days No Yes No No Yes No 
R1 Inventory 1-2 weeks No N/A No No No No 
11 Inventory <4 weeks Yes Yes No No Yes No 
G1 Inventory 1-2 months No N/A No No No No 
C1 Rapid response 3-5 days U U No No U U 
D1 Rapid response <1 week (could be <1 day) | No Yes No No No No 
A2 Rapid response 15 days No N/A No No Yes No 
D3 Rapid response 2-4 weeks No Yes No No No No 
F1 Rapid response 4 weeks Yes Yes U [Yes] Yes Sometimes 
U1 Rapid response 1 month (minimum) U N/A U U Sometimes No 
T1 Rapid response 2-6 weeks Yes Yes No No U U 
12 Rapid response 4-8 weeks Yes Yes No No Yes No 
R2 Rapid response 1-3 months Yes Yes No No U No 
O1 Rapid response 10-12 weeks Yes N/A No N/A Yes No 
Bi Rapid review 3 days Yes N/A No No Yes No 
E2 Rapid review 4 weeks Yes Yes No No Yes No 
H1 Rapid review 1 month Yes U No No U U 
A3 Rapid review 30 days Yes Yes No No Yes No 
A4 Rapid review 30 days Yes Yes No No Yes Yes 
Ji Rapid review 4-6 weeks (up to 16 max) Yes Yes Yes [Yes] Yes Yes 
C2 Rapid review 4-8 weeks (up to 10) Yes Yes No No No No 
S1 Rapid review 2 person-months U U U U U Yes 
13 Rapid review 8-12 weeks Yes Yes No No Yes No 
L2 Rapid review 2-4 months Yes No No No U Yes 
L3 Rapid review 2-4 months Yes Yes Yes No Yes Yes 
G2 Rapid review 3-4 months Yes Yes Yes [Yes] Yes Yes 
02 Rapid review 3-4 months Yes Yes Sometimes Sometimes Yes Sometimes 
P1 Rapid review 3-4 months Sometimes Yes U U Yes No 
R3 Rapid review 3-6 months U U No No U Yes 
M1 Rapid review 3-8 months Yes Yes Partialt [Yes] U Yes 
M2 Rapid review 3-6 months (NS) Yes Yes Partialt [Yes] U Yes 
N1 Rapid review 3-6 months (NS) Yes Sometimes Yes [Yes] Yes No 
D2 Algorithm 5 minutes No Yes No No No No 
K1 Algorithm 1-6 weeks Yes No Yes Yes Yes Sometimes 





* - working code used for internal purposes; + - second person reviews excludes at title/abstract stage, dual review of full text 
N/A — no data extraction 

[Yes] — one with verification 
NS = not specified; interviewee did not provide more detail on timeframe 
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Table D-4. Approach to analysis, synthesis, and conclusions in different rapid review products 




















































































































a aes : Quality/ROB SOE/GRADE Qualitative Quantitative , 
Classification Timeframe appraisal eonducied analysis analysis Conclusions 
E1 Inventory 3-5 days No No No No No 
Al Inventory 5-10 days No No No No Key messages 
L1 Inventory 5-15 days No No No No Opinion 
R1 Inventory 1-2 weeks No No No No No 
i Inventory <4 weeks No No No No No 
G1 Inventory 1-2 months No No No No No 
C1 Rapid response | 3-5 days No No No No No 
D1 Rapid response | <1 week (could No No No No No 
be <1 day) 
A2 Rapid response | 15 days No No No No Summary of best evidence 
D3 Rapid response | 2-4 weeks No No No No Report what studies concluded - no 
explicit review conclusion 
F1 Rapid response | 4 weeks Yes Sometimes No No Yes 
U1 Rapid response | 1 month No U No No Key messages 
(minimum) 
T1 Rapid response | 2-6 weeks No No Yes No No 
l2 Rapid response | 4-8 weeks Yes No Yes No Summary of what best guidelines, 
protocols, and systematic reviews say 
R2 Rapid response | 1-3 months No No No No Summary of best evidence 
O1 Rapid response | 10-12 weeks No No Yes No [Yes] 
Bi Rapid review 3 days Yes Yes Yes No Yes 
E2 Rapid review 4 weeks Yes Yes Yes No Yes 
H1 Rapid review 1 month U Yes Yes No Yes 
A3 Rapid review 30 days Yes No Yes No Yes 
A4 Rapid review 30 days Yes No Yes No Yes 
Ji Rapid review 4-6 weeks (up to | Yes [No] Yes No Yes 
16 max) 
C2 Rapid review 4-8 weeks (up to | Yes Yes Yes No Yes 
10 
$1 Rapid review 2 person-months | Yes No Yes Yes No 
13 Rapid review 8-12 weeks Yes Yes Yes No No 
L2 Rapid review 2-4 months Yes Yes Yes Yes [Yes] 
L3 Rapid review 2-4 months Yes No Yes No [Yes] 
G2 Rapid review 3-4 months Yes Yes Yes Yes Yes 
02 Rapid review 3-4 months Yes U Yes No Yes 
P41 Rapid review 3-4 months Sometimes Sometimes Sometimes Sometimes Opinion 
R3 Rapid review 3-6 months No No Yes No Yes 
M1 Rapid review 3-8 months Yes Yes Yes Yes Yes 
M2 Rapid review 3-6 months (NS) _| Yes Yes Yes Yes [Yes] 
N14 Rapid review 3-6 months (NS) _| Yes Yes Yes Yes Yes 
D2 Algorithm 5 minutes No No Yes Yes Yes 
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z ae a : Quality/ROB SOE/GRADE Qualitative Quantitative : 
Classification Timeframe appraisal éonductad analysis analysis Conclusions 
K1 Algorithm 1-6 weeks Yes Yes No No No 


























* - working code used for internal purposes 
[No] — under consideration 
[Yes] - yes conclusions, no recommendations 


NS = not specified; interviewee did not provide more detail on timeframe 


U = unknown 
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Appendix E. Types of Rapid Review Products 
Described by Key Informants 


This appendix provides additional information that was gathered from the Key Informant 
interviews. Some of this content overlaps with the information that is presented in the results’ 
section of the main report based on the combined analysis of the Key Informant interviews and 
the literature. 

Interviewees described several rapid review products, some completed within days (Table E- 
1). Tables E1-4 below describe the products and the names that Key Informants used to describe 
them according to the timeframe in which they were produced. Most organizations appeared to 
have started with a product that was produced in several months. Because rapid reviewers have a 
strong relationship with a particular requestor, they often spoke about asking the requestor when 
they needed to make the decision, and then the reviewer would work back to present what could 
be done in that timeframe and explore whether this would be helpful. Over time, various 
products were developed in the back-and-forth conversation between reviewers and requestors 
ranging from products that were completed within days to months. 


Table E-1. Products completed in days (based on Key Informant interviews) 





























Name of 

product 

Reference 1-7 days List of available evidence, Work directly with requestor to determine what is needed. 

List sorted by study design. Search major databases (Embase, PubMed, Medline, 
Cochrane) and other HTA agencies. 

Scoping 3-5 days List of available evidence. Patient submitted appeal for single technology or 

Review intervention. Search TRIP, PubMed, Cochrane. Send list 
to committee for decisionmaking. 

Rapid 3 days Narrative summary of Receive questions from health plans and work directly 

Review evidence with with nominator. Search Medline, Embase and summarize 

recommendation. available evidence. Includes quality appraisal and 

grading. 

Rapid 7 days or 1 page narrative summary | Work directly with clinicians who submit questions. 

Review less of evidence. Search first for existing reviews (TRIP, Cochrane) and go 
to primary evidence if necessary. 





Interviewees also described several rapid review products that use existing reviews as the 
primary or sole source of evidence. These products were narrative summaries of the existing 
evidence. Details of these products are outlined in Table E-2. 


Table E-2. Products primarily using existing reviews (based on Key Informant interviews) 




















Name of 
product 

1 week or 1 page Work directly with clinicians who submit questions. Search first for 

less narrative existing reviews (TRIP, Cochrane) and go to primary evidence if 
summary of necessary. 
evidence. 

1 month Narrative Work directly with requestors to determine questions/timeframe. 
summary of Search their own database (DARE) which has quality rated existing 
existing reviews. If no reviews are found, do not search primary evidence but 
evidence. tell requestor that no existing reviews are available. 





Table E-3 provides an overview of the different rapid products and Table E-4 provides a list 
of the end-users by timeframe; the information in these tables is based on the Key Informant 


interviews. 
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Table E-3. Landscape of rapid products by timeframe (based on Key Informant interviews) 
Timeframe Landscape of reports 

<7 days There are 4 different reports, the scopes of these were varied but were fairly narrow (single 
intervention, evaluating effectiveness only, single technology). The styles of these reports varied 
from a reference list to presenting the scope of the evidence to narratives. Only one included 
internal review, one study considered feedback on the website as external review. None of the 
reports performed quantitative analysis, though one did conduct qualitative analysis. One of the 
four provided recommendations. The number of people working on the reports ranged from 2-10. 
One type relied on existing SRs. 
1-2 weeks Two reports were conducted in 1 to 2 weeks. Dual review was not conducted for either of these, 
nor did they rely solely on existing SRs. Neither report conducted dual extraction nor did they 
conduct quality appraisal, and only one conducted a full text review. Internal or external review 
was not conducted and no recommendations were provided. 
1-2 months Sixteen reports were conducted and most of these were classified as Rapid Reviews. Others 
were Brief Reports; Rapid Summary; Evidence Briefing; Medical Technology Assessment; 
Literature Summary; Health Technology Assessment; Evidence Inventory; Evidence-based 
Briefs and Evidence Advisory. The purpose for most of these was to be used in health system 
decisions. Very few of the reports conducted dual study selection. Some used existing SRs but 
not as the sole source of evidence. Almost all of the data extraction was conducted by just one 
person, most conducted quality appraisal for primary studies as well as SRs using AMSTAR. 
Some conducted strength of evidence assessments and the most used method was GRADE. 
Most of these reports had a structured format, many included evidence tables and a narrative 
synthesis. Few reports did not conduct any sort of internal or external review; most included 
external review. Most of the reports conducted qualitative synthesis, and some conducted both 
qualitative and quantitative synthesis. Very few of the reports provided guidance. 1-10 people 

could have worked on any given report. 

3-6 months Eight reports were conducted in 3-6 months most of which are referred to as rapid reviews. 
Literature selection was mostly done by one person. All of these reviews included existing SRs, 
not as sole data source. Most conducted data extraction by one person, quality appraisal and 
SOE. Most of the reports had a structured format and the styles of the reports varied but most 
included external review. None conducted a quantitative analysis, some did a qualitative 
analysis, and none of the reports provided guidance. 
12-18 months One report was conducted in 12-18 months. This is a single technology appraisal that produces 
national guidance to inform drug funding. The reviewers did not conduct a literature search as 
this was already done by the manufacturer. The manufacturer provided most of the report pieces 


(search, template, synthesis) and the organization acted as a dual reviewer in the process. 


























Table E-4. End-users of rapid review products by timeframe 
Timeframe 





























